Using the Generic Document Profile to Cluster Similar Texts

The World Wide Web contains a huge quantity of text that is notoriously inefficient to use. This work aims to apply a text processing technique based on thesaurally derived lexical chains to improve Internet Information Retrieval where a lexical chain a set of words in a text that are related by both proximity, and by relations derived from an external lexical knowledge source such as WordNet, Roget's Thesaurus, LDOCE, and so on. Finding Information on the Internet is notoriously hard, even when users have a clear focus to their queries. This situation is exacerbated when users only have vague notions about the topics they wish to explore. This could be remedied using Exemplar Texts, where an Exemplar Text is the ideal model result for Web searches. Our problem is now transformed into one of identifying similar texts. The Generic Document Profile is designed to allow the comparison of document similarity whilst being independent of terminology and document length. It is simply a set of semantic categories derived from Roget's thesaurus with associated weights. These weights are based on lexical chain length and strength. A Generic Document Profile can be compared to another using a Case Based Reasoning approach. Case Based Reasoning (CBR) is a problem solving method that seeks to solve existing problems by reference to previous successful solutions. Here our Exemplar Texts count as previous solutions (and in these experiments, MS Encarta is used as the source of Exemplar Texts). In CBR a query and examples are usually represented as attribute value pairs. Thus to apply CBR to document comparison, both the text acting as a query, and the documents to be compared against need to be represented in equivalent terms. If this representation is based on simple terms (ie word stems), the problem becomes hugely complex, since there are many words in a language. This would also be a fragile approach, since semantically equivalent words word not count as equal. However, If we represent a document as Roget categories, the problem becomes tractable. We use Roget's thesaurus since there are 1024 main thesaurus categories as opposed to 50000+ Synsets in Wordnet. This approach is providing interesting results, although it is subject to the well known Word Sense Disambiguation problem. The paper will summarise current findings, cover some implementation details, and describe to approach taken to full-text Word Sense Disambiguation. Hesperus is a system designed to address this problem. Using an electronic encyclopedia as a source of subject defining texts, queries are made to MetaCrawler. This searches several search engines in parallel returning the best 10-20 links. Hesperus retrieves these web pages, and computes their conceptual similarity to the Exemplar Text using a method based on thesaurally determined lexical chains. Initial results show that users prefer the Hesperus page order to MetaCrawler's statistical ordering. The technique consequently shows promise as a way to improve the effectiveness of Web searching.